The present invention relates to apparatus and methods for optical image acquisition and analysis. In particular, it relates to passive techniques for measuring the ranges of objects and their locations in three dimensions.
In many surveying and related applications, it is desirable to rapidly measure the locations of all of the visible objects in a scene in three dimensions. Conventional passive image acquisition and processing techniques are effective for determining the bearings of objects, but do not adequately provide range information.
Various active techniques are used for determining the range of objects, including direct measurement, radar, sonar, scanned laser and structured light methods. Direct measurement involves holding rulers and the like up to the objects of interest or reference locations and reading the scale. This usually requires two people, and can be error-prone, time consuming and sometimes dangerous. Measurements in three dimensions can be difficult. Other active techniques involve transmitting energy to the object and monitoring the reflection of that energy. These methods have several shortcomings. They often fail when the object does not reflect the transmitted energy well or when the ambient energies are too high. Production of the transmitted energy requires special hardware that consumes power and is often expensive and failure prone. When several systems are operating in close proximity, the possibility of mutual interference exists. Scanned laser systems can be slow. Sonar is prone to errors caused by wind. Most of these active systems do not produce enough information to identify objects.
Range information can be obtained using a conventional camera, if the object or the camera is moving a known way. The motion of the image in the field of view is compared with motion expected for various ranges in order to infer the range. However, the method is useful only in limited circumstances.
Another approach is known as photogrammetry. Distinctive markers are placed within the scene, and photographs are taken from a number of angles. The locations of the markers in the resulting images are measured and triangulation is performed to infer the camera locations and, eventually, locations of objects of interest. This method is very time-consuming and tedious.
Other approaches make use of passive optical techniques. These generally break down into stereo and focus methods. Stereo methods mimic human stereoscopic vision, using images from two cameras to estimate range. Stereo methods can be very effective, but they suffer from a problem in aligning parts of images from the two cameras. In cluttered or repetitive scenes, such as those containing soil or vegetation, the problem of determining which parts of the images from the two cameras to align with each other can be intractable. Image features such as edges that are coplanar with the line segment connecting the two lenses cannot be used for stereo ranging.
Focus techniques can be divided into autofocus systems and range mapping systems. Autofocus systems are used to focus cameras at one or a few points in the field of view. They measure the degree of blur at these points and drive the lens focus mechanism until the blur is minimized. While these can be quite sophisticated, they do not produce point-by-point range mapping information that is needed in some applications.
In focus-based range mapping systems, multiple cameras or multiple settings of a single camera are used to make several images of the same scene with differing focus qualities. Sharpness is measured across the images and point-by-point comparison of the sharpness between the images is made in a way that effects of the scene contrast cancel out. The remaining differences in sharpness indicate the distance of the objects at the various points in the images.
Systems of this last type are described by Pentland, who used two or more cameras with differing apertures to obtain simultaneous images. A bulky beamsplitter/mirror apparatus is placed in front of the cameras to ensure that they have the same view of the scene. This multiple camera system is too costly, heavy, and limited in power to find widespread use. A variation is described in U.S. Pat. No. 5,365,597, where a camera system includes dual camera optics in which a beamsplitter is used within the lens system to simplify the optical design. Another improvement of Pentland""s multiple camera method is described by Nourbakhsh et al. (U.S. Pat. No. 5,793,900). Nourbakhsh et al. describe a system using three cameras with different focus distance settings, rather than different apertures as in Pentland""s presentation. This system allows for rapid calculation of ranges, but sacrifices range resolution in order to do so. The use of multiple sets of optics tends to make the camera system heavy and expensive. It is also difficult to synchronize the optics if overall focus, zoom, or iris need to be changed. The beamsplitters themselves must be large since they have to be sized to full aperture and field of view of the system. Moreover, the images formed in this way will not be truly identical due to manufacturing variations between the sets of optics.
An alternative method that uses only a single camera is described by Nakagawa et al. in U.S. Pat. No. 5,151,609. This approach is intended for use with a microscope. In this method, the object under consideration rests on a platform that is moved in steps toward or away from the camera. A large number of images can be obtained in this way, which increases the range-finding power relative to Pentland""s method. In a related variation, the camera and the object are kept fixed and the focus setting of the lens is changed step-wise. Even in a static situation, such as a surveying application, the time to complete the measurement can be excessive. Even if the scene and the camera location and orientation are static, the acquisition of multiple images by changing the camera settings is time consuming and introduces problems of control, measurement, and recording of the camera parameters to associate with the images. Also, changing the focus setting of a lens may cause the image to shift laterally if the lens rotates during the focus change and the optical axis and the rotation axis are not in perfect alignment.
Thus, it would be desirable to provide a simplified method by which ranges and locations of objects can be determined easily and accurately. In particular, it would be desirable to provide a method by which range-mapping for substantially all objects in the field of view of a camera can be provided easily and accurately. It is further desirable to perform these measurements using relatively simple, portable equipment. It would also be desirable to produce a detailed photographic record of a scene, from which location measurements can be made as their need becomes apparent.
In one aspect, this invention is a camera system comprising
a) a camera having a camera line of sight, an image sensor and a means for capturing images formed on the image sensor;
b) a focussing system for focussing light to form an image on the image sensor;
c) an optical displacement unit (ODU) that receives light from an offset line of sight that is different from the camera line of sight and directs the light into the camera along or parallel to the camera line of sight, the ODU being rotatable about a rotation axis that is the same as or parallel to the camera line of sight but different than the offset line of sight,
wherein all elements of the focussing system, except for an optional elements for increasing field of view, are located between the ODU and the image sensor.
In a second aspect, this invention is a camera system comprising
a) a camera having a camera line of sight, an image sensor and a means for capturing images formed on the image sensor;
b) a focussing system for focussing light to form an image on the image sensor; and
c) an optical displacement unit (ODU) that receives light from an offset line of sight that is different from the camera line of sight and directs the light into the camera along or parallel to the camera line of sight, the ODU being rotatable about a rotation axis that is the same as or parallel to the camera line of sight but different than the offset line of sight, wherein the ODU is adapted to operate while physically detached from the camera.
In another aspect, this invention is a method for determining the range of an object, comprising
a) bringing a scene containing the object within the field of view of a camera system including (1) a camera having a camera line of sight, an image sensor, a means for capturing images formed on the image sensor, (2) a focussing system for focussing light to form an image of the scene on the image sensor and (3) an optical displacement unit (ODU) that receives light from an offset line of sight that is different from the camera line of sight and directs the light into the camera along the camera line of sight, the ODU being rotatable about a rotation axis that is the same as or substantially parallel to the camera line of sight but different than the offset line of sight, so as to form an image of the scene on the image sensor;
b) capturing said image of the scene to produce a first frame;
c) rotating the optical displacement unit about the rotation axis to a different position while maintaining the orientation of the image sensor substantially constant with respect to said object, so as to effect an apparent motion of the image on the image sensor;
d) capturing another image to produce a subsequent frame;
e) determining the position of an image corresponding to at least one point on said object on the first frame and at least one subsequent frame; and
f) calculating the range of the object from the positions of the image corresponding to said at least one point on said object on said first frame and at least one subsequent frame.
This aspect of the invention provides a method by which ranges of one or more objects within the field of view of the camera can be made easily and quickly. By repeating steps c)-f) for all objects in the field of view of the camera system, a range map is built that provides range information for most or all of the objects in the scene. Furthermore, steps c)-f) can be repeated one or more times, through a full rotation of the optical displacement unit if desired, to provide a larger number of frames. The range calculation in step f) can be made from position information from all (or some subset of) the frames to provide more accurate and complete range calculations.
In a third aspect, this invention is a method for determining the range of an object comprising
a) bringing a scene containing the object within the field of view of a camera having (1) a camera having a camera line of sight, an image sensor and a means for capturing images formed on the image sensor, (2) a focussing system for focussing light to form an image on the image sensor and (3) an optical displacement unit (ODU) that receives light along an offset line of sight that is different from the camera line of sight and directs the light into the camera along the camera line of sight, the ODU being rotatable about a rotation axis that is the same as or substantially parallel to the camera line of sight but different than the offset line of sight, to form an image of the scene thereon,
b) capturing said image of the scene to produce a first frame,
c) rotating the optical displacement unit about the rotation axis to a different position while maintaining the orientation of the image sensor substantially constant with respect to said object, so as to effect an apparent motion of the image on the image sensor;
d) capturing another image to produce a subsequent frame,
e) repeating steps c) and d) one or more times to produce additional subsequent frames;
f) identifying a given distance;
g) shifting each of said first and subsequent frames to produce a corresponding number of shifted images, such that objects at said given distance appear at substantially the same pixel locations in each of said shifted images;
h) summing said shifted images to form a composite image in which objects at said given distance appear to be in focus; and
i) identifying pixels in said composite image having objects in focus, and assigning said given distance to objects imaged on said pixels in said composite image.
Again, the method provides an easy and accurate method of determining ranges of objects. As discussed further below, variations of this method permit increased resolution images to be formed so that improved accuracy in location estimates can be obtained. Another variation permits the dynamic range of the image to be improved, again allowing for improved accuracy of the method. Moreover, the method of the invention provides for easy and rapid determination of the transverse coordinates of an object. Together with the range determination, this permits three-dimensional mapping of any object within the field of view of the camera system.